Pandas Profiling Report

Dataset statistics

Number of variables	9
Number of observations	45312
Missing cells	0
Missing cells (%)	0.0%
Duplicate rows	0
Duplicate rows (%)	0.0%
Total size in memory	5.4 MiB
Average record size in memory	124.2 B

Variable types

NUM	8
CAT	1

Reproduction

Analysis started	2020-03-22 12:10:16.512411
Analysis finished	2020-03-22 12:10:51.793781
Version	pandas-profiling v2.5.0
Command line	`pandas_profiling --config_file config.yaml [YOUR_FILE.csv]`
Download configuration	config.yaml

`vicprice` is highly skewed (γ1 = 78.68700693)	Skewed
`period` has 944 (2.1%) zeros	Zeros

date
Real number (ℝ_≥0)

Distinct count	933
Unique (%)	2.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	0.4990795572
Minimum	0
Maximum	1
Zeros	48
Zeros (%)	0.1%
Memory size	354.1 KiB

Quantile statistics

Minimum	0
5-th percentile	0.005133
Q1	0.031934
median	0.456329
Q3	0.88054725
95-th percentile	0.907969
Maximum	1
Range	1
Interquartile range (IQR)	0.84861325

Descriptive statistics

Standard deviation	0.3403077218
Coefficient of variation (CV)	0.6818706895
Kurtosis	-1.318137405
Mean	0.4990795572
Median Absolute Deviation (MAD)	0.2825371897
Skewness	-0.1658145258
Sum	22614.2929
Variance	0.1158093455

Histogram
Dynamic Histogram

Histogram with fixed size bins (bins=10)

Histogram with variable size bins (bins=[0.000000e+00 2.200000e-05 1.040000e-03 4.181000e-03 5.420000e-03 ... 9.156450e-01 9.157780e-01 9.711075e-01 9.932305e-01 1.000000e+00], "bayesian blocks" binning strategy used)

Value	Count	Frequency (%)
0.898102	96	0.2%
0.875979	96	0.2%
0.442326	96	0.2%
0.871731	96	0.2%
0.473563	96	0.2%
0.876156	96	0.2%
0.867307	96	0.2%
0.880404	96	0.2%
0.893677	96	0.2%
0.889253	96	0.2%
Other values (923)	44352	97.9%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	48	0.1%
4.4e-05	48	0.1%
8.8e-05	48	0.1%
0.000133	48	0.1%
0.000177	48	0.1%

Value	Count	Frequency (%)
1	48	0.1%
0.995575	48	0.1%
0.995443	48	0.1%
0.991018	48	0.1%
0.986594	48	0.1%

day
Real number (ℝ_≥0)

Distinct count	7
Unique (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	4.003177966
Minimum	1
Maximum	7
Zeros	0
Zeros (%)	0.0%
Memory size	354.1 KiB

Quantile statistics

Minimum	1
5-th percentile	1
Q1	2
median	4
Q3	6
95-th percentile	7
Maximum	7
Range	6
Interquartile range (IQR)	4

Descriptive statistics

Standard deviation	1.998694937
Coefficient of variation (CV)	0.4992770629
Kurtosis	-1.248878713
Mean	4.003177966
Median Absolute Deviation (MAD)	1.713374838
Skewness	-0.001187819335
Sum	181392
Variance	3.994781452

Histogram
Dynamic Histogram

Histogram with fixed size bins (bins=10)

Histogram with variable size bins (bins=[1. 1.5 6.5 7. ], "bayesian blocks" binning strategy used)

Value	Count	Frequency (%)
7	6480	14.3%
6	6480	14.3%
5	6480	14.3%
4	6480	14.3%
3	6480	14.3%
2	6480	14.3%
1	6432	14.2%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
1	6432	14.2%
2	6480	14.3%
3	6480	14.3%
4	6480	14.3%
5	6480	14.3%

Value	Count	Frequency (%)
7	6480	14.3%
6	6480	14.3%
5	6480	14.3%
4	6480	14.3%
3	6480	14.3%

period
Real number (ℝ_≥0)

ZEROS

Distinct count	48
Unique (%)	0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	0.5
Minimum	0
Maximum	1
Zeros	944
Zeros (%)	2.1%
Memory size	354.1 KiB

Quantile statistics

Minimum	0
5-th percentile	0.042553
Q1	0.25
median	0.5
Q3	0.75
95-th percentile	0.957447
Maximum	1
Range	1
Interquartile range (IQR)	0.5

Descriptive statistics

Standard deviation	0.2947563828
Coefficient of variation (CV)	0.5895127656
Kurtosis	-1.201042076
Mean	0.5
Median Absolute Deviation (MAD)	0.2553190833
Skewness	8.506244433e-17
Sum	22656
Variance	0.0868813252

Histogram
Dynamic Histogram

Histogram with fixed size bins (bins=10)

Histogram with variable size bins (bins=[0. 0.0106385 0.9893615 1. ], "bayesian blocks" binning strategy used)

Value	Count	Frequency (%)
0.638298	944	2.1%
0.276596	944	2.1%
0.553191	944	2.1%
0.085106	944	2.1%
0.382979	944	2.1%
0.595745	944	2.1%
0.680851	944	2.1%
0.489362	944	2.1%
0.617021	944	2.1%
0.808511	944	2.1%
Other values (38)	35872	79.2%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	944	2.1%
0.021277	944	2.1%
0.042553	944	2.1%
0.06383	944	2.1%
0.085106	944	2.1%

Value	Count	Frequency (%)
1	944	2.1%
0.978723	944	2.1%
0.957447	944	2.1%
0.93617	944	2.1%
0.914894	944	2.1%

nswprice
Real number (ℝ_≥0)

Distinct count	4089
Unique (%)	9.0%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	0.05786831014
Minimum	0
Maximum	1
Zeros	29
Zeros (%)	0.1%
Memory size	354.1 KiB

Quantile statistics

Minimum	0
5-th percentile	0.02681
Q1	0.035127
median	0.048652
Q3	0.074336
95-th percentile	0.106611
Maximum	1
Range	1
Interquartile range (IQR)	0.039209

Descriptive statistics

Standard deviation	0.03999076895
Coefficient of variation (CV)	0.69106509
Kurtosis	158.5274187
Mean	0.05786831014
Median Absolute Deviation (MAD)	0.02328855723
Skewness	9.073261555
Sum	2622.128869
Variance	0.001599261602

Histogram
Dynamic Histogram

Histogram with fixed size bins (bins=10)

Histogram with variable size bins (bins=[0.000000e+00 3.000000e-05 1.022300e-02 1.258000e-02 1.343500e-02 ... 1.792210e-01 1.796565e-01 2.311010e-01 3.982825e-01 1.000000e+00], "bayesian blocks" binning strategy used)

Value	Count	Frequency (%)
0.074817	383	0.8%
0.055242	272	0.6%
0.058394	266	0.6%
0.028942	219	0.5%
0.041732	200	0.4%
0.041431	175	0.4%
0.028822	152	0.3%
0.029422	151	0.3%
0.044134	150	0.3%
0.041642	149	0.3%
Other values (4079)	43195	95.3%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	29	0.1%
6e-05	1	< 0.1%
0.000901	2	< 0.1%
0.001831	1	< 0.1%
0.001861	1	< 0.1%

Value	Count	Frequency (%)
1	1	< 0.1%
0.981806	2	< 0.1%
0.979975	1	< 0.1%
0.970908	1	< 0.1%
0.968296	1	< 0.1%

nswdemand
Real number (ℝ_≥0)

Distinct count	5266
Unique (%)	11.6%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	0.4254178953
Minimum	0
Maximum	1
Zeros	1
Zeros (%)	< 0.1%
Memory size	354.1 KiB

Quantile statistics

Minimum	0
5-th percentile	0.143707
Q1	0.309134
median	0.4436925
Q3	0.536001
95-th percentile	0.684023
Maximum	1
Range	1
Interquartile range (IQR)	0.226867

Descriptive statistics

Standard deviation	0.1633227142
Coefficient of variation (CV)	0.3839112459
Kurtosis	-0.4111293582
Mean	0.4254178953
Median Absolute Deviation (MAD)	0.1328403352
Skewness	-0.09773906855
Sum	19276.53567
Variance	0.02667430897

Histogram
Dynamic Histogram

Histogram with fixed size bins (bins=10)

Histogram with variable size bins (bins=[0. 0.036522 0.053481 0.0823415 0.0978135 ... 0.777819 0.820217 0.881583 0.9283695 1. ], "bayesian blocks" binning strategy used)

Value	Count	Frequency (%)
0.488099	33	0.1%
0.478578	32	0.1%
0.487057	31	0.1%
0.451056	30	0.1%
0.489586	30	0.1%
0.480512	29	0.1%
0.517852	29	0.1%
0.4759	29	0.1%
0.473222	29	0.1%
0.50967	28	0.1%
Other values (5256)	45012	99.3%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	1	< 0.1%
0.00119	1	< 0.1%
0.001488	1	< 0.1%
0.003868	1	< 0.1%
0.008182	1	< 0.1%

Value	Count	Frequency (%)
1	1	< 0.1%
0.980809	1	< 0.1%
0.962511	1	< 0.1%
0.960875	1	< 0.1%
0.956561	1	< 0.1%

vicprice
Real number (ℝ_≥0)

SKEWED

Distinct count	3798
Unique (%)	8.4%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	0.003467033898
Minimum	0
Maximum	1
Zeros	35
Zeros (%)	0.1%
Memory size	354.1 KiB

Quantile statistics

Minimum	0
5-th percentile	0.001525
Q1	0.002277
median	0.003467
Q3	0.003467
95-th percentile	0.00586
Maximum	1
Range	1
Interquartile range (IQR)	0.00119

Descriptive statistics

Standard deviation	0.01021303821
Coefficient of variation (CV)	2.945756665
Kurtosis	7047.774288
Mean	0.003467033898
Median Absolute Deviation (MAD)	0.001110384583
Skewness	78.68700693
Sum	157.09824
Variance	0.0001043061495

Histogram
Dynamic Histogram

Histogram with fixed size bins (bins=10)

Histogram with variable size bins (bins=[0.00000e+00 2.90000e-05 5.09000e-04 5.75500e-04 6.59500e-04 ... 1.34085e-02 1.93275e-02 3.03330e-02 6.63650e-02 1.00000e+00], "bayesian blocks" binning strategy used)

Value	Count	Frequency (%)
0.003467	17427	38.5%
0.000841	223	0.5%
0.002009	130	0.3%
0.000839	90	0.2%
0.000976	89	0.2%
0.000837	78	0.2%
0.001973	59	0.1%
0.001853	54	0.1%
0.001793	53	0.1%
0.001959	52	0.1%
Other values (3788)	27057	59.7%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	35	0.1%
5.8e-05	3	< 0.1%
0.000118	3	< 0.1%
0.000129	2	< 0.1%
0.000139	1	< 0.1%

Value	Count	Frequency (%)
1	1	< 0.1%
0.996228	1	< 0.1%
0.980682	1	< 0.1%
0.837239	1	< 0.1%
0.509112	1	< 0.1%

vicdemand
Real number (ℝ_≥0)

Distinct count	2846
Unique (%)	6.3%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	0.4229150757
Minimum	0
Maximum	1
Zeros	1
Zeros (%)	< 0.1%
Memory size	354.1 KiB

Quantile statistics

Minimum	0
5-th percentile	0.210772
Q1	0.372346
median	0.422915
Q3	0.46925175
95-th percentile	0.641119
Maximum	1
Range	1
Interquartile range (IQR)	0.09690575

Descriptive statistics

Standard deviation	0.1209653455
Coefficient of variation (CV)	0.2860275087
Kurtosis	0.9089617644
Mean	0.4229150757
Median Absolute Deviation (MAD)	0.07964014647
Skewness	0.1595328979
Sum	19163.12791
Variance	0.01463261481

Histogram
Dynamic Histogram

Histogram with fixed size bins (bins=10)

Histogram with variable size bins (bins=[0. 0.02913 0.0789745 0.100207 0.1484985 ... 0.7580265 0.7971255 0.841792 0.8907305 1. ], "bayesian blocks" binning strategy used)

Value	Count	Frequency (%)
0.422915	17425	38.5%
0.562144	28	0.1%
0.494562	28	0.1%
0.542206	28	0.1%
0.333506	28	0.1%
0.328068	28	0.1%
0.51709	27	0.1%
0.340497	27	0.1%
0.506732	26	0.1%
0.287675	26	0.1%
Other values (2836)	27641	61.0%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	1	< 0.1%
0.024081	1	< 0.1%
0.025634	1	< 0.1%
0.028742	1	< 0.1%
0.029518	1	< 0.1%

Value	Count	Frequency (%)
1	1	< 0.1%
0.996893	1	< 0.1%
0.991196	1	< 0.1%
0.989384	1	< 0.1%
0.983687	1	< 0.1%

transfer
Real number (ℝ_≥0)

Distinct count	1878
Unique (%)	4.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	0.5005263909
Minimum	0
Maximum	1
Zeros	1
Zeros (%)	< 0.1%
Memory size	354.1 KiB

Quantile statistics

Minimum	0
5-th percentile	0.29760945
Q1	0.414912
median	0.414912
Q3	0.605702
95-th percentile	0.801754
Maximum	1
Range	1
Interquartile range (IQR)	0.19079

Descriptive statistics

Standard deviation	0.1533733911
Coefficient of variation (CV)	0.3064241845
Kurtosis	-0.08657400985
Mean	0.5005263909
Median Absolute Deviation (MAD)	0.1263068164
Skewness	0.6705411339
Sum	22679.85183
Variance	0.02352339711

Histogram
Dynamic Histogram

Histogram with fixed size bins (bins=10)

Histogram with variable size bins (bins=[0. 0.074781 0.1217105 0.16557 0.19057 ... 0.902851 0.9039475 0.9282895 0.9458335 1. ], "bayesian blocks" binning strategy used)

Value	Count	Frequency (%)
0.414912	17480	38.6%
0.500526	338	0.7%
0.765789	220	0.5%
0.838158	75	0.2%
0.634211	68	0.2%
0.67807	58	0.1%
0.841667	57	0.1%
0.662719	56	0.1%
0.857018	56	0.1%
0.72193	54	0.1%
Other values (1868)	26850	59.3%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	1	< 0.1%
0.000439	1	< 0.1%
0.002632	1	< 0.1%
0.004386	1	< 0.1%
0.00614	1	< 0.1%

Value	Count	Frequency (%)
1	1	< 0.1%
0.959211	1	< 0.1%
0.957895	1	< 0.1%
0.946053	1	< 0.1%
0.945614	1	< 0.1%

class
Categorical

Distinct count	2
Unique (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Memory size	354.1 KiB

DOWN	26075
UP	19237

Value	Count	Frequency (%)
DOWN	26075	57.5%
UP	19237	42.5%

Length

Max length	4
Mean length	3.150909251
Min length	2

Value	Count	Frequency (%)
Uppercase_Letter	6	100.0%

Value	Count	Frequency (%)
Latin	6	100.0%

Value	Count	Frequency (%)
ASCII	6	100.0%

Pearson's r

The Pearson's correlation coefficient (r) is a measure of linear correlation between two variables. It's value lies between -1 and +1, -1 indicating total negative linear correlation, 0 indicating no linear correlation and 1 indicating total positive linear correlation. Furthermore, r is invariant under separate changes in location and scale of the two variables, implying that for a linear function the angle to the x-axis does not affect r.

To calculate r for two variables X and Y, one divides the covariance of X and Y by the product of their standard deviations.

Spearman's ρ

The Spearman's rank correlation coefficient (ρ) is a measure of monotonic correlation between two variables, and is therefore better in catching nonlinear monotonic correlations than Pearson's r. It's value lies between -1 and +1, -1 indicating total negative monotonic correlation, 0 indicating no monotonic correlation and 1 indicating total positive monotonic correlation.

To calculate ρ for two variables X and Y, one divides the covariance of the rank variables of X and Y by the product of their standard deviations.

Kendall's τ

Similarly to Spearman's rank correlation coefficient, the Kendall rank correlation coefficient (τ) measures ordinal association between two variables. It's value lies between -1 and +1, -1 indicating total negative correlation, 0 indicating no correlation and 1 indicating total positive correlation.

To calculate τ for two variables X and Y, one determines the number of concordant and discordant pairs of observations. τ is given by the number of concordant pairs minus the discordant pairs divided by the total number of pairs.

Count
Matrix

First rows

	day	period	nswprice	nswdemand	vicprice	vicdemand	transfer	class
0	2	0.000000	0.056443	0.439155	0.003467	0.422915	0.414912	UP
1	2	0.021277	0.051699	0.415055	0.003467	0.422915	0.414912	UP
2	2	0.042553	0.051489	0.385004	0.003467	0.422915	0.414912	UP
3	2	0.063830	0.045485	0.314639	0.003467	0.422915	0.414912	UP
4	2	0.085106	0.042482	0.251116	0.003467	0.422915	0.414912	DOWN
5	2	0.106383	0.041161	0.207528	0.003467	0.422915	0.414912	DOWN
6	2	0.127660	0.041161	0.171824	0.003467	0.422915	0.414912	DOWN
7	2	0.148936	0.041161	0.152782	0.003467	0.422915	0.414912	DOWN
8	2	0.170213	0.041161	0.134930	0.003467	0.422915	0.414912	DOWN
9	2	0.191489	0.041161	0.140583	0.003467	0.422915	0.414912	DOWN

Last rows

	date	day	period	nswprice	nswdemand	vicprice	vicdemand	transfer	class
45302	0.9158	7	0.808511	0.055062	0.350193	0.003772	0.295961	0.365351	DOWN
45303	0.9158	7	0.829787	0.065420	0.353913	0.004508	0.319524	0.319737	UP
45304	0.9158	7	0.851064	0.055902	0.340077	0.003857	0.313568	0.375000	DOWN
45305	0.9158	7	0.872340	0.050648	0.322821	0.003488	0.305541	0.325877	DOWN
45306	0.9158	7	0.893617	0.058875	0.340226	0.004049	0.276541	0.351754	DOWN
45307	0.9158	7	0.914894	0.044224	0.340672	0.003033	0.255049	0.405263	DOWN
45308	0.9158	7	0.936170	0.044884	0.355549	0.003072	0.241326	0.420614	DOWN
45309	0.9158	7	0.957447	0.043593	0.340970	0.002983	0.247799	0.362281	DOWN
45310	0.9158	7	0.978723	0.066651	0.329366	0.004630	0.345417	0.206579	UP
45311	0.9158	7	1.000000	0.050679	0.288753	0.003542	0.355256	0.231140	DOWN

Overview

Variables

Interactions

Correlations

Pearson's r

Spearman's ρ

Kendall's τ

Missing values

Sample

First rows

Last rows